Endoscopic tutorial system

ABSTRACT

A system for simulating a medical procedure performed on a subject, featuring a simulated organ, a simulated medical instrument and a locator for determining the location of the instrument in the organ. The system further features a visual display for displaying images from the medical procedure. The visual display also includes a three-dimensional mathematical model for modeling the organ, which is divided into a plurality of linear segments. The location of the instrument in the organ is used to select the segment, which in turn is used to select the images for display on the visual display.

FIELD AND BACKGROUND OF THE INVENTION

The present invention relates to a system and method for teaching andtraining students in medical procedures, and in particular to a systemand method for training students in the procedure of endoscopy.

Endoscopy, and in particular flexible gastro-endoscopy, are examples ofminimally invasive medical procedures. Flexible gastro-endoscopy is animportant medical tool for both surgical and diagnostic procedures inthe gastro-intestinal tract. Essentially, gastro-endoscopy is performedby inserting an endoscope, which is a flexible tube, into thegastro-intestinal tract, either through the mouth or the rectum of thesubject. The tube is manipulated by a trained physician throughspecialized controls. The end of the tube which is inserted into thesubject contains a camera and one or more surgical tools, such as aclipper for removing tissue samples from the gastro-intestinal tract.The physician must maneuver the tube according to images of thegastro-intestinal tract received from the camera and displayed on avideo screen. The lack of direct visual feedback from thegastro-intestinal tract is one factor which renders endoscopy a complexand difficult procedure to master. Such lack of feedback also increasesthe difficulty of hand-eye coordination and correct manipulation of theendoscopic device. Thus, flexible gastro-endoscopy is a difficultprocedure to both perform and to learn.

Currently, students are taught to perform flexible gastro-endoscopyaccording to the traditional model for medical education, in whichstudents observe and assist more experienced physicians. Unfortunately,such observation alone cannot provide the necessary training for suchcomplicated medical procedures. Students may also perform procedures onanimals and human cadavers, neither of which replicates the visual andtactile sensations of a live human patient. Thus, traditional medicaltraining is not adequate for modem technologically complex medicalprocedures.

In an attempt to provide more realistic medical training for suchprocedures, simulation devices have been developed which attempt toreplicate the tactile sensations and/or visual feedback for theseprocedures, in order to provide improved medical training withoutendangering human patients. An example of such a simulation device isdisclosed in U.S. Pat. No. 5,403,191, in which the disclosed device is abox containing simulated human organs. Various surgical laparoscopicprocedures can be performed on the simulated organs. Visual feedback isprovided by a system of mirrors. However, the system of both visual andtactile feedback is primitive in this device, and does not provide atrue representation of the visual and tactile sensations which wouldaccompany such surgical procedures in a human patient. Furthermore, thebox itself is not a realistic representation of the three-dimensionalstructure of a human patient. Thus, the disclosed device is lacking inmany important aspects and fails to meet the needs of a medicalsimulation device.

Attempts to provide a more realistic experience from a medicalsimulation devices are disclosed in PCT Patent Application Nos. WO96/16389 and WO 95/02233. Both of these applications disclose a devicefor providing a simulation of the surgical procedure of laparoscopy.Both devices include a mannequin in the shape of a human torso, withvarious points at which simulated surgical instruments are placed.However, the devices are limited in that the positions of the simulatedsurgical instruments are predetermined, which is not a realisticscenario. Furthermore, the visual feedback is based upon a stream ofvideo images taken from actual surgical procedures. However, such simplerendering of video images would result in inaccurate or unrealisticimages as portions of the video data would need to be removed forgreater processing speed. Alternatively, the video processing wouldconsume such massive amounts of computational time and resources thatthe entire system would fail to respond in a realistic time period tothe actions of the student. At the very minimum, a dedicated graphicsworkstation would be required, rather than a personal computer (PC).Thus, neither reference teaches or discloses adequate visual processingfor real time visual feedback of the simulated medical procedure.

Similarly, U.S. Pat. No. 4,907,973 discloses a device for simulating themedical procedure of flexible gastro-endoscopy. The disclosed devicealso suffers from the deficiencies of the above-referenced prior artdevices, in that the visual feedback system is based upon rendering ofvideo data taken from actual endoscopic procedures. As noted previously,displaying such data would either require massive computationalresources, or else would simply require too much time for a realisticvisual feedback response. Thus, the disclosed device also suffers fromthe deficiencies of the prior art.

A truly useful and efficient medical simulation device for minimallyinvasive therapeutic procedures such as endoscopy would give real time,accurate and realistic visual feedback of the procedure, and would alsogive realistic tactile feedback, so that the visual and tactile systemswould be accurately linked for the simulation as for an actual medicalprocedure. Unfortunately, such a simulation device is not currentlytaught or provided by the prior art.

There is therefore a need for, and it would be useful to have, a methodand a system to simulate a minimally invasive medical procedure such asendoscopy, which would provide accurate, linked visual and tactilefeedback to the student and which would serve as a training resource forall aspects of the procedure.

SUMMARY OF THE INVENTION

The present invention includes a method and a system to simulate theminimally invasive medical procedure of endoscopy, particularly offlexible gastro-endoscopy. The system is designed to simulate the actualmedical procedure of endoscopy as closely as possible by providing botha simulated medical instrument, and tactile and visual feedback as thesimulated procedure is performed on the simulated patient.

According to the present invention, there is provided a system forperforming a simulated medical procedure, comprising: (a) a simulatedorgan; (b) a simulated instrument for performing the simulated medicalprocedure on the simulated organ; (c) a locator for determining alocation of the simulated instrument within the simulated organ; and (d)a visual display for displaying images according to the location of thesimulated instrument within the simulated organ for providing visualfeedback, such that the images simulate actual visual data receivedduring an actual medical procedure as performed on an actual subject,the visual display including: (i) a mathematical model for modeling thesimulated organ according to a corresponding actual organ, the modelbeing divided into a plurality of segments; (ii) a loader for selectingat least one of the plurality of segments for display, the at least oneof the plurality of segments being selected according to the location ofthe simulated instrument within the simulated organ; (iii) a controllerfor selecting a simulated image from the segment according to thelocation of the simulated instrument; and (iv) a displayer fordisplaying the simulated image.

Preferably, the visual displayer further comprises: (v) a texturemapping database for storing texture mapping data; and (vi) a texturemapping engine for overlaying the simulated image with the texturemapping data substantially before the simulated image is displayed bythe displayer. More preferably, the texture mapping is animation ofrandom movement of the simulated instrument and random movement of thesimulated organ.

Also preferably, the texture mapping includes images obtained fromperforming the actual medical procedure on the actual subject.

More preferably, the images are obtained by first recording the visualdata during the performance and then selecting the images from therecorded visual data.

According to a preferred embodiment of the present invention, themathematical model features a plurality of polygons constructedaccording to a spline, the spline determining a geometry of themathematical model in three dimensions. Preferably, a deformation in themathematical model corresponding to a deformation in the simulated organis determined by altering the spline. More preferably, the deformationin the simulated organ is a local deformation, the local deformation ofthe simulated organ being determined according to the mathematical modelby adding polygons to a portion of the mathematical model, such that theportion of the mathematical model is deformed to produce the localdeformation. Most preferably, the mathematical model is constructed fromthe spline by modeling the simulated organ as a straight line andaltering the spline until the mathematical model fits the correspondingactual organ. Also most preferably, the controller selects the simulatedimage according to at least one previous movement of the simulatedinstrument ok within the simulated organ.

According to other preferred embodiments of the present invention, thedisplayer further displays a graphical user interface. Preferably, thegraphical user interface displays tutorial information for aid inperforming the medical procedure.

According to still other preferred embodiments of the present invention,the simulated organ is a gastrointestinal tract. Preferably, thegastro-intestinal tract is constructed from a semi-flexible, smoothmaterial. Also preferably, the simulated instrument is an endoscope, theendoscope featuring a sensor for determining a location of the sensor inthe gastro-intestinal tract, the system further comprising: (e) acomputer for determining the visual feedback according to the locationof the sensor.

Preferably, the system also features a tactile feedback mechanism forproviding simulated tactile feedback according to the location of thetip of the endoscope.

According to one embodiment of the tactile feedback mechanism, thetactile feedback mechanism is contained in the gastro-intestinal tract,and the gastrointestinal tract further comprises: (i) a plurality ofservo-motors; (ii) a piston operated by each of the plurality ofservo-motors, the piston contacting the semi-flexible material; and(iii) a controller for controlling the plurality of servo-motors, suchthat a position of the piston is determined by the controller, and suchthat the position of the piston provides the tactile feedback.

Alternatively, the tactile feedback mechanism is located in theendoscope, and the endoscope further comprises: (i) a guiding sleeveconnected to the tip of the endoscope; (ii) at least one ball bearingattached to the guiding sleeve for rolling along an inner surface of thegastrointestinal tract; (iii) at least one linear motor attached to theguiding sleeve; (iv) a piston operated by the linear motor, the pistoncontacting the inner surface of the gastro-intestinal tract; and (v) acontroller for controlling the linear motor, such that a position of thepiston is determined by the controller, and such that the position ofthe piston provides the tactile feedback.

Also alternatively, the tactile feedback mechanism features: (i) aplurality of rings surrounding the endoscope, each ring having adifferent radius, at least a first ring featuring a radius greater thana radius of the endoscope and at least a second ring featuring a radiusless than the radius of the endoscope, the radius of each of theplurality of rings being controlled according to a degree of inflationwith air of each of the plurality of rings, the radius of the ringsdetermining movement of the endoscope; (ii) an air pump for pumping airinto the plurality of rings; (iii) at least one tube for connecting theair pump to the plurality of rings; and (iv) an air pump controller fordetermining the degree of inflation with air of the plurality of ringsby controlling the air pump.

Preferably, the at least one tube is two tubes, a first tube for pumpingair into the plurality of rings and a second tube for suctioning airfrom the plurality of rings, and the air pump pumps air into theplurality of rings and sucks air from the plurality of rings, such thatthe degree of inflation with air of the plurality of rings is determinedby alternately pumping air into, and suctioning air from, the pluralityof rings.

Also preferably, the gastro-intestinal tract is a substantially straighttube, such that the tactile feedback and the visual feedback aresubstantially independent of a geometrical shape of the gastrointestinaltract. Preferably, the tactile feedback mechanism is operated accordingto tactile feedback obtained during the performance of the medicalprocedure on an actual subject, the tactile feedback being obtainedthrough virtual reality gloves.

According to other preferred embodiments of the system of the presentinvention, the endoscope further features a handle for holding theendoscope and a tool unit, the tool unit comprising: (i) a simulatedtool; (ii) a channel for receiving the simulated master of an actualtool, such as forceps or snare, the channel being located in the handle;(iii) a tool control unit for detecting a movement of the simulatedtool, the tool control unit being located in the channel and the toolcontrol unit being in communication with the computer, such that thecomputer determines the visual feedback and the tactile feedbackaccording to the movement of the simulated tool.

Preferably, the tool control unit detects a location of the simulatedtool within the gastrointestinal tract for providing visual feedback.

More preferably, the tool control unit additionally detects a roll ofthe simulated tool for providing visual feedback.

According to one embodiment of the tool control unit, the tool controlunit further comprises: (1) a light source for producing light, thelight source being located in the channel; (2) a light wheel foralternately blocking and unblocking the light according to the movementof the simulated tool; and (3) a light detector for detecting the light,such that the computer determines a movement of the simulated toolaccording to the light detector.

According to another embodiment of the present invention, there isprovided a method for performing a simulated endoscopic procedure,comprising the steps of: (a) providing a system for performing thesimulated endoscopic procedure, comprising: (i) a simulatedgastro-intestinal tract; (ii) a simulated endoscope for performing thesimulated endoscopic procedure on the simulated gastrointestinal tract;(iii) a locator for determining a location of the simulated endoscopewithin the simulated gastro-intestinal tract; and (iv) a visual displayfor displaying images according to the simulated endoscope within thesimulated gastro-intestinal tract, such that the images simulate visualdata received during an actual medical procedure as performed on anactual subject, the visual display including: (1) a three-dimensionalmathematical model of the simulated gastro-intestinal tract, the modelbeing divided into a plurality of segments; (2) a loader for selectingat least one of the plurality of segments for display, the at least oneof the plurality of segments being selected according to the location ofthe simulated endoscope within the simulated gastrointestinal tract; (3)a controller for selecting a simulated image from the segment accordingto the location of the simulated instrument, and (4) a displayer fordisplaying the simulated image according to the controller, such thatthe simulated image is a displayed image; (b) inserting the simulatedendoscope into the simulated gastro-intestinal tract; (c) receivingvisual feedback according to the displayed image; and (d) receivingtactile feedback according to the location of the endoscope within thegastrointestinal tract.

Preferably, the displayed image is determined according to at least oneprevious movement of the simulated endoscope within the simulatedgastrointestinal tract.

According to yet another embodiment of the present invention, there isprovided a method for displaying simulated visual data of a medicalprocedure performed on an actual human organ with an actual medicalinstrument, the method comprising the steps of: (a) recording actualdata from a performance of an actual medical procedure on a living humanpatient; (b) abstracting a plurality of individual images from theactual data; (c) digitizing the plurality of individual images to form aplurality of digitized images; (d) selecting at least one of theplurality of digitized images to form a selected digitized image; (e)storing the selected digitized image as texture mapping data in atexture mapping database; (f) providing mathematical model of the actualhuman organ, the model being divided into a plurality of segments; (g)selecting one of the plurality of segments from the model for display;(h) overlaying the texture mapping data from the texture mappingdatabase onto the segment of the model to form at least one resultantimage; and (i) displaying the resultant image.

Preferably, the actual data from the performance of the actual medicalprocedure is selected from the group consisting of video data, MRI(magnetic resonance imaging) data and CAT (computer assisted tomography)scan data.

More preferably, step (f) further comprises the steps of: (i) modelingthe actual human organ as a plurality of polygons according to a spline;(ii) mapping the spline to the actual human organ according tothree-dimensional coordinates; (iii) altering the spline such that thespline fits the actual data.

Most preferably, the texture mapping data further include animation.Also most preferably, the animation includes random movement of theactual medical instrument and random movement of the actual human organ.

According to still another embodiment of the present invention, there isprovided a method for teaching a particular skill required forperformance of an actual medical procedure to a student, the actualmedical procedure being performed with an actual medical instrument onan actual organ with visual feedback, the method comprising the stepsof: (a) providing a simulated instrument for simulating the actualmedical instrument; (b) providing a simulated organ for simulating theactual organ; (c) abstracting a portion of the visual feedback of theactual medical procedure; (d) providing the portion of the visualfeedback for simulating the visual feedback; and (e) manipulating thesimulated instrument within the simulated organ by the student accordingto the portion of the visual feedback, such that a motion of thesimulated instrument is the skill taught to the student.

Preferably, the portion of the visual feedback includes substantiallyfewer visual details than the visual feedback of the actual medicalprocedure.

More preferably, the simulated organ is a simulation of agastro-intestinal tract, and the simulated instrument is a simulation ofan endoscope.

Most preferably, the portion of the visual feedback includes only ageometrical shape of an interior of the gastrointestinal tract.

The method of the present invention for preparing a model of thesimulated organ, and for rendering the visual feedback of the simulatedorgan during the simulated medical procedure, can be described as aplurality of instructions being performed by a data processor. As such,these instructions can be implemented in hardware, software or firmware,or a combination thereof. As software, the steps of the method of thepresent invention could be implemented in substantially any suitableprogramming language which could easily be selected by one of ordinaryskill in the art, including but not limited to, C and C++.

Hereinafter, the term “simulated medical procedure” refers to thesimulation of the medical procedure as performed through the system andmethod of the present invention. Hereinafter, the term “actual medicalprocedure” refers to the performance of the medical procedure on anactual, living human patient with an actual endoscope, such that themedical procedure is “real” rather than “simulated”. Hereinafter, theterm “corresponding actual organ” refers to the “real” organ of a humanbeing or other mammal which is being simulated by the simulated organ ofthe present invention.

Hereinafter, the term “endoscopy” includes, but is not limited to, theprocedure of flexible gastro-endoscopy, as previously described, andmedical diagnostic and surgical procedures in which an endoscope isinserted into the mouth or the rectum of the subject for manipulationwithin the gastro-intestinal tract of the subject. Hereinafter, the term“subject” refers to the human or lower mammal upon which the method andsystem of the present invention are performed or operated. Hereinafter,the term “student” refers to any human using the system of the presentinvention, being trained according to the present invention or beingtaught according to the present invention including, but not limited to,students attending medical school or a university, a medical doctor, atrained gastro-enterologist or other trained medical specialist.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, aspects and advantages will be betterunderstood from the following detailed description of a preferredembodiment of the invention with reference to the drawings, wherein:

FIG. 1 is an exemplary illustration of the system for medical simulationaccording to the present invention;

FIG. 2 is an exemplary illustration of a screen display according to thepresent invention;

FIG. 3A is a flowchart of an exemplary method according to the presentinvention for preparation of the visual model of the simulated organ andrendering of visual feedback and

FIG. 3B is a schematic block diagram of an exemplary visual processingand display system according to the present invention;

FIG. 4 is a schematic block diagram of an exemplary tutorial systemaccording to the present invention;

FIGS. 5A and 5B illustrate an exemplary simulated gastro-intestinaltract according to the present invention;

FIGS. 6A-C illustrate various aspects of one embodiment of theforce-feedback system according to the present invention;

FIGS. 7A-7D illustrate a second embodiment of the force-feedback systemaccording to the present invention;

FIGS. 8A-8E show another embodiment of the system according to thepresent invention; and

FIGS. 9A-9E show an illustrative embodiment of a tool unit according tothe present invention.

BRIEF DESCRIPTION OF THE INVENTION

The present invention includes a method and a system to simulate themedical procedure of endoscopy, particularly of flexiblegastro-endoscopy. The system is designed to simulate the actual medicalprocedure of endoscopy as closely as possible by providing both asimulated medical instrument, and tactile and visual feedback as thesimulated procedure is performed on the simulated patient. Although thediscussion is directed toward the medical procedure of endoscopy, thepresent invention could also be employed to simulate other types ofminimally invasive medical procedures.

The system of the present invention features both a physical model and avirtual model for the simulation of the medical procedure of endoscopy.The physical model includes a mannequin into which the simulatedendoscope is inserted. A simulated organ is located within themannequin. For example, if the simulated organ is the gastrointestinaltract, the organ may optionally include a simulated rectum and asimulated colon for simulating the procedure of flexiblegastro-endoscopy. Optionally and preferably, the simulated organ mayoptionally include a simulated mouth and upper gastro-intestinal tract.The simulated endoscope is inserted into the simulated gastro-intestinaltract. The simulated gastro-intestinal tract includes a tactile feedbacksystem for providing realistic tactile feedback according to themovement of the simulated endoscope within the simulated organ.

The virtual model provides a “virtual reality” for the simulation ofimages from the endoscope. In an actual endoscopic medical procedure, acamera at the tip of the actual endoscope returns images from thegastro-intestinal tract of the human patient. These images are thenviewed by the physician performing the endoscopic procedure, therebyproviding visual feedback to the physician. The system of the presentinvention provides a “virtual reality” for the realistic simulation ofthis visual feedback. This virtual reality enables the real-time displayof realistic images of the gastro-intestinal tract on a video monitoraccording to the manipulations of the simulated endoscope, preferably insuch a manner that the tactile and visual feedback are linked as theywould be in a human patient.

The virtual reality has two main components: a three-dimensional,mathematical model of the gastrointestinal tract, or a portion thereof,and a database of enhanced digitized images derived from actual visualdata obtained from actual endoscopic procedures. These two componentsare combined to provide realistic visual feedback by using the enhancedimages as texture mapping to overlay the mathematical model of thesimulated organ, thereby closely simulating images obtained from theactual procedure.

The virtual reality feedback of the gastro-intestinal tract isparticularly advantageous for simulating images because it does not relyon video streams, which require massive computational power forreal-time display of visual feedback. In addition, video streams provideonly a predetermined flow of images and cannot provide visual data withsix degrees of freedom in real time. Furthermore, the virtual reality ofthe present invention does not rely merely on a mathematical model ofthe gastrointestinal tract, which cannot capture the irregularities andsubtle visual features of an actual gastrointestinal tract from a humanpatient. Thus, the virtual reality feedback of the gastrointestinaltract provides the best simulation of realistic images in real time forvisual feedback.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The present invention is related to a method and a system to simulatethe procedure of endoscopy, particularly of flexible gastro-endoscopy.The system includes a mannequin into which the simulated endoscope isinserted. Visual feedback is provided through a video monitor, whichdisplays realistic images in real time, according to the manipulationsof the simulated endoscope. Realistic tactile feedback is also provided,preferably in such a manner that the tactile and visual feedback arelinked as they would be in a human patient. Preferably, the presentinvention also features a tutorial system for training students andtesting their performance. Thus, the system and method of the presentinvention provide a realistic simulation of the medical procedure ofendoscopy for training and testing students.

The principles and operation of a method and a system according to thepresent invention for medical simulation, and in particular for thesimulation of the medical procedure of endoscopy, preferably includingcommunicating tutorial results and measurement of student skills to theteacher or supervising medical personnel, may be better understood withreference to the drawings and the accompanying description, it beingunderstood that these drawings are given for illustrative purposes onlyand are not meant to be limiting. Furthermore, although the descriptionbelow is directed toward the simulation of the colon, it should be notedthat this is only for the purposes of clarity and is not meant to belimiting in any way.

Referring now to the drawings, FIG. 1 depicts an exemplary, illustrativesystem for medical simulation according to the present invention. Asystem 10 includes a mannequin 12 representing the subject on which theprocedure is to be performed, a simulated endoscope 14 and a computer 16with a video monitor 18. A student 20 is shown interacting with system10 by manipulating simulated endoscope 14 within mannequin 12. Asfurther illustrated in FIGS. 5A and 5B below, mannequin 12 includes asimulated organ into which simulated endoscope 14 is inserted. Asstudent 20 manipulates simulated endoscope 14, tactile and visualfeedback are determined according to the position of endoscope 14 withinthe simulated organ (not shown). The visual feedback are provided in theform of a display on video monitor 18. The necessary data calculationsare performed by computer 16, so that realistic tactile and visualfeedback are provided to student 20.

FIG. 2 is an exemplary illustration of a screen display shown on monitor18. A screen display 22 includes a feedback image 24. Feedback image 24represents the visual image as seen if the endoscope were inserted intoa living human patient. Feedback image 24 is an accurate and realisticsimulation of the visual data that would be received from that portionof the gastrointestinal tract in the living human patient. Althoughfeedback image 24 is shown as a static image, it is understood that thisis for illustrative purposes only and the actual visual feedback datawould be in the form of a substantially continuous flow of simulatedimages based upon actual video stream data obtained from an actualendoscopic procedure. Thus, the flow of images represented by feedbackimage 24 gives the student (not shown) realistic visual feedback.

In addition, screen display 22 preferably includes a number of GUI(graphic user interface) features related to the preferred tutorialfunctions of the present invention. For example, a tracking display 26explicitly shows the location of the simulated endoscope within thesimulated gastrointestinal tract. Tracking display 26 includes aschematic gastro-intestinal tract 28, into which a schematic endoscope30 has been inserted. Preferably, tracking display 26 can be enabled ordisabled, so that the student can only see tracking display 26 if thetracking function is enabled.

Additional, optional but preferred features of screen display 22 includethe provision of a “help” button 32, which upon activation could causethe display of such helpful information as a guide to the controls ofthe endoscope. Similarly, a preferred “hint” button 34 would give thestudent one or more suggestions on how to continue the performance ofthe medical procedure. A preferred “patient history” button 36 wouldcause screen display 22 to show information related to one of aselection of simulated “patient histories”, which could be of help tothe student in deciding upon a further action. Finally, a preferred“performance” button 38 would cause screen display 22 to display areview and rating of the performance of the student. All of thesefunctions are part of the preferred embodiment of a tutorial system fortraining a student in the medical procedure of endoscopy, as describedin further detail in FIG. 4.

FIGS. 3A and 3B are schematic block diagrams of an exemplary visualprocessing and display system and method according to the presentinvention. FIG. 3A is a flow chart of the method for visual processingand display according to the present invention, and is intended as asummary of the method employed by the system of FIG. 3B. Further detailsconcerning particular aspects of the method are described below withreference to FIG. 3B.

The method and system of the present invention provide a solution to anumber of problems in the art of medical simulation, in particular forthe simulation of the procedure of gastro-endoscopy. This procedureinvolves the visual display of an interior portion of thegastrointestinal tract, such as the colon. The colon is a flexible bodywith a curved structure. The inner surface of the colon is generallydeformable, as well as being specifically, locally deformable. All ofthese deformations in space must be calculated according to themathematical model of the colon, and then rendered visually in real timein order to provide a realistic visual feedback response for the user.

FIG. 3A shows a preferred embodiment of the method of the presentinvention for preparation of the model and rendering of visual feedback,including steps required for preparation of the computerized model ofthe colon, as well as steps required for display of the colon.

In step 1 of the method of the present invention, actual video data arerecorded onto videotape during the performance of the actual medicalprocedure of endoscopy on a living human patient. In addition, such datacould also include MRI (magnetic resonance imaging) and CAT (computerassisted tomography) scan data from procedures performed on living humanpatients.

In step 2, individual images are abstracted, for example with aframegrabber device, and then digitized. In step 3, the digitized imagesare preferably selected for clarity and lack of visual artifacts, andare then stored in a texture mapping database. More preferably, thedigitized images are enhanced before being stored. Most preferably, thetexture mapping also include animation. Such animation could simulateeffects such as random vibration of the tissue of the colon or of theendoscope, as well as such events as liquid flowing downward due to theinfluence of gravity.

In step 4, a three-dimensional mathematical model of the human colon isconstructed. The three-dimensional mathematical model of the colon whichis particularly preferred for the present invention is a polygonal modelsuch as a spline. This mathematical function represents the colon as aseries of curves, such that the points in the three-dimensionalstructure of the colon are mapped to the spline. For example, the coloncould be modeled as a straight line which is deformed by altering thespline for the model until the model fits the data. Alternatively, thespline could be placed inside the colon and mapped to the colon.Preferably, multiple splines are used to model the junction of thestomach and small intestine, for example.

The mapping can be performed according to three-dimensional coordinates,along the x, y and z axes. Alternatively, the mapping can be performedaccording to coordinates of time, angle and radius within the colon. Amixture of these two different types of coordinates is also optionallyemployed, in which the coordinates are time, x and y for example. Boththe spline itself and the mapping from the spline to the colon canoptionally be altered in order to provide new and different visualrepresentations of the colon, for example in order to provide aplurality of theoretical “test cases” for students to study. Thealteration is optionally performed according to MRI (magnetic resonanceimaging) data, for example. In addition, optionally and preferably datafrom MRI and/or CAT scan procedures are cleaned and reassembledaccording to the mathematical model, in order to more accuratelydetermine the geometry of the simulated colon. Substantially all ofthese procedures could be performed automatically according to such dataor alternatively, these procedures could also be performed partially orwholly manually. Thus, the preferred mathematical model of the presentinvention permits the data to be rapidly visually rendered onto themodel of the colon.

According to a particularly preferred embodiment of the presentinvention, a “loop” of the endoscope cable itself is modeled. Such aloop occurs when the person performing the endoscopic procedure, whether“real” or simulated, inadvertently changes direction within the colon byturning the endoscope itself. Such a loop can be very dangerous to thepatient, and therefore should be detected as part of a simulation, inorder to warn the student as an indication that the procedure has beenperformed incorrectly thereby causing the loop to appear.

Preferably, the loop is constructed with a spline according to thepresent invention and is coordinated with force feedback. The length ofcable which has been fed into colon must be determined, as must thelength of the colon from the rectum (entry point of the endoscope) tothe current position of the endoscope. The size of the loop is thencalculated from the differential of these two lengths, and the loop ismodeled according to the spline.

The method of visually rendering the colon according to the presentinvention includes a number of steps, described below, which areperformed as software instructions operated by a data processor. Themethod preferably includes the step (shown as step 5 in FIG. 3A) ofdividing the colon into a plurality of portions. The division is madelinearly, since the spatial movement of the simulated endoscope islimited. In other words, the simulated endoscope cannot “jump” from oneportion of the colon to another, but must instead proceed in a linearfashion along the simulated colon. In addition, the simulated endoscopecan only be moved at a finite speed through the simulated colon. Thus,the endoscope must pass through each segment of the three-dimensionalmodel of the colon in sequence at a known, limited speed.

The consequences of such a division is that only one segment needs to beprocessed in any given moment, although a plurality of such segmentscould be processed substantially simultaneously if the computingresources were available. Furthermore, the division reduces the visualprocessing into a much more manageable task, since this model mayoptionally include thousands of polygons in the preferred embodiment,although each segment has far fewer polygons.

In addition, preferably only those portions which are in the line ofsight of the camera, and hence either immediately visible or soon tobecome visible, are selected for visual rendering in order to decreasethe computations required for the rendering. More preferably, the numberof portions which are rendered is not predetermined, since under certaincircumstances, the number of portions in the line of sight may vary. Forexample, when the camera is traveling around a bend in the colon, theline of sight of the camera is very short, such that relatively fewerportions, or else smaller such portions, must be rendered.

Next, in step 6, the visual attributes of the area of the colon beingscanned by the camera are determined. Preferably, these visualattributes are determined according to a number of factors, includingthe location of the tip of the endoscope, which holds the camera, andthe direction in which the camera itself is pointed. Other importantfactors include the shape of the colon being modeled and the history ofmovement of the camera through the colon. With regard to the latterfactor, the previous movements of the endoscope through the colon, asdetermined by the actions of the student, have a significant impact onthe area of the colon which is visualized by the camera at any givenmoment. For example, if the student has caused a “loop” to form byincorrectly operating the endoscope, as previously described, this“loop” can be simulated correctly only through the inclusion of thehistory of movements to determine the visual feedback.

In step 7, preferably a local deformation to at least one of theseportions is analyzed to determine if such a deformation affects thespline itself. The mapped coordinates are then rapidly transformed fromtime, angle and radius to x, y and z. Next, in step 8 preferably thelocal deformation of the tissue of the colon is determined throughinterpolation of the radius, in order to determine the degree of suchdeformation. Since the time, angle and radius may not give sufficientinformation to perform this calculation, optionally and preferably, thevolume of the colon is additionally altered according to predefinedmathematical models.

For deformations on a highly local scale, such as the point of contactbetween the tip of the endoscopic instrument and the colon at a lowdegree of force from the instrument, preferably the level of details inthe area is increased by adding more polygons to the calculationsperformed with the model in order to be able to stretch all orsubstantially points in the immediate area without distortion. Thestretching is preferably performed according to a predetermined functionwhich preferably enables the spline model to be altered locally.

This preferred method for modeling “stretching” of the colon can also beused to model local areas of irregularity such as a polyp. Polyps can bemapped point by point onto the model of the colon, thereby adjusting thevisual representation of the tissue to accommodate both the polyp itselfand the structural alterations of the tissue at the base of the polyp.

Next, in step 9, the various types of data which were previouslydescribed are used to actually render the visual data onto the colon.Initially, the mapping of such data onto the model optionally andpreferably involves some adjustments, performed manually by a softwareprogrammer. Alternatively, such mapping could be entirely automaticallyperformed.

In step 10, texture mapping from the database is overlaid onto the chunkof the model. Preferably, such texture mapping includes both thedigitized images and additional animation. In step 11, the resultantimages are displayed. As noted previously, the images are displayed in acontinuous flow according to the location of the simulated endoscopewithin the simulated gastrointestinal tract. Also as noted previously,such mapping of coordinates is preferably performed according to themathematical model of the colon, which is more preferably a spline.

FIG. 3B shows the visual processing and display system according to thepresent invention in more detail. A visual processing and display system40 includes screen display 22 for displaying the processed visual data.The visual data are constructed as follows. First, data are recordedfrom actual gastro-endoscopic procedures onto videotape, as shown in arecording block 42. The data are preferably stored on Super-VHFvideotape in order to obtain the highest quality representation of thevisual images displayed on the screen during the actual endoscopicprocedure, as shown in block 44. Next, at least a portion of the framesof the videotape, and preferably substantially all the frames, areabstracted individually by a frame-grabber 46 to form digitized images.Individual digitized images can then be selected for clarity and lack ofartifacts such as reflections from the endoscopic apparatus itself. Theimages in the selected frames are then preferably enhanced and added toa texture mapping database 48.

Preferably, two types of texture mapping are stored in the database. Thefirst type of texture mapping is intended to enhance the realisticvisual aspects of the images, for example by removing visual artifacts.The second type of texture mapping is intended to simulate the behaviorof a live organ and a real endoscope, as represented by block 50. Duringactual endoscopic procedures on a living human patient, the tissue ofthe colon moves somewhat, and the endoscope itself vibrates and wobbles.This movement is simulated visually by the addition of random animationof the images, and also by the addition of such effects as liquidflowing downward due to the influence of gravity. Such animationenhances the realistic nature of the visual representation of the colon.

In order for the enhanced images to be correctly displayed, the imagesmust correspond to the manipulation and location of the simulatedendoscope within the simulated colon. In particular, the texture mappingof the images should correspond to the location of the endoscope withinthe colon. Such correspondence between the location of the endoscopewithin the colon and the texture mapping is provided by a texturemapping engine 52. The texture mapping data is then readily accessed bythe display portion of visual system 40, as shown by block 54.

However, as noted for previous prior art devices, simply reproducing theselected enhanced frames in a massive video stream would quicklyoverwhelm the computational resources and cause the visual display tobecome unsynchronized from the physical location of the simulatedendoscope. Furthermore, such a video stream would not enable the correctdisplay of images according to the movement of the endoscope, whichpreferably has six degrees of freedom. Thus, mere reproduction is notsufficient to ensure realistic images, even when mapped onto athree-dimensional surface.

Preferably, visual processing and display system 40 includes athree-dimensional mathematical model of at least a portion of thegastro-intestinal tract 56, more preferably constructed as described inFIG. 3A. For the purposes of discussion, model 56 is herein described asa three-dimensional model of the colon, it being understood that this isnot meant to be limiting in any way. Model 56 preferably features aplurality of segments 58, more preferably many such segments 58.

As the simulated endoscope moves along the simulated colon, the locationof the endoscope is given to a locator 60, described in further detailbelow. Locator 60 then instructs an object loader 62 to load therelevant segment 58 for access by visual system 40, as shown in block 54and previously described. In the preferred embodiment shown, preferablythree segments 58 are ready for access by object loader 62 at any givenmoment. The specific segment 58 in which the endoscope is currentlylocated is preferably held in DRAM or RAM, in combination with thetexture mapping described previously. The next segment 58 and thepreceding segment 58 preferably are also stored in an easily accessiblelocation, although not necessarily in RAM or DRAM.

Preferably, the display of each image from specific segment 58 intowhich the simulated endoscope has entered is optimized by a segmentoptimizer 64. Segment optimizer 64 receives information from locator 60,as well as the series of images obtained from overlaying the texturemapping onto the relevant segment 58, and then feeds each specific imageto a display manager 66 for display on screen display 22.

In addition, display manager 66 is assisted by a real-time viewer 68,preferably implemented in Direct 3D™ (Microsoft Inc., Seattle, Wash.).Real-time viewer 68 provides the necessary software support tocommunicate with a graphics card 70 for actual display of the images onscreen display 22. Although graphics card 70 can be of any suitablemanufacture, preferably graphics card 70 has at least 8, and morepreferably at least 16, Mb of VRAM for optimal performance. An exampleof a suitable graphics card 70 is the 3Dfx Voodoo Rush™ card.Preferably, the performance of real-time viewer 68 is enhanced by a mathoptimizer 72, preferably implemented in Visual C++. The interactionbetween segment optimizer 64 and display manager 66 on the one hand, andlocator 60 on the other, is provided through a software interface 74.Software interface 74 enables locator 60 to communicate with the othercomponents of visual system 40, in order to provide informationregarding the location of the endoscope within the colon.

In preferred embodiments of the present invention, locator 60 includes asensor 76, which can be obtained from Ascension Technology Corp., forexample. Sensor 76 senses positional information from within a simulatedorgan 77, which is described herein as a colon for the purposes ofdiscussion and is not meant to be limiting. Sensor 76 is controlled by acontrol unit 82. The positional information is then relayed to a CPUcontroller 78, which is connected to a servo-motor 80 (Haydon Switch andInstrument Co.). As the simulated endoscope moves through the colon, theendoscope contacts different portions of the colon (not shown; see FIGS.5 and 6 below). Tactile feedback is provided by each servo-motor 80 inturn, which manipulates the material of the colon.

All Visual system 40 also includes a user interface 84, preferablyimplemented in Visual C++. User interface 84 includes the GUI featuresdescribed previously for FIG. 2. In addition, user interface 84 enablesvisual system 40 to interact with the preferred feature of a networkinterface 86, for example, so that other students can view screendisplay 22 over a network. User interface 84 also permits the tutorialfunctions of at least one, and preferably a plurality of, tutorialmodules 88 to be activated. Tutorial module 88 could include aparticular scenario, such as a subject with colon cancer, so thatdifferent types of diagnostic and medical challenges could be presentedto the student. The student would then need to respond correctly to thepresented scenario.

An example of the tutorial system is illustrated in more detail in theblock diagram of FIG. 4. A tutorial system 90 starts as shown in block92. Next, the user must select whether actual interaction with thesimulated endoscope is desired, or if the user prefers to receivetutoring in the theory of endoscopy, as shown in a block 94. The nextdisplay asks if the user is new, as shown in a block 96. If the answeris “yes”, the user is requested to enter certain information, as shownby block 98. If the answer is “no”, the user is requested to enteridentification information, such as user name or identification number,as shown in block 100.

Next, the user must select the type of tutoring. For example, the usercould select tutoring by subject 102, tutoring by procedures 104 ortutoring by case studies 106. Tutoring by subject 102 includes, but isnot limited to, such subjects as basic manipulation of the endoscope,biopsy and polypectomy. Tutoring by subject 102 includes on-screensupport, as shown in block 108.

Tutoring by case studies 106 can be selected both according to casenumber and according to the level of the desired cases, such asbeginner, intermediate and expert. Preferably, individual case studiescould be created by a teacher or professor, by combining features ofvarious stored cases. For example, a professor could create a casehistory appropriate for a 20 year old male with colitis, so that thestudent would then be able to practice endoscopy on such a patient.Thus, tutoring system 90 preferably has the flexibility to enable manydifferent types of “patients” to be studied.

If desired, on-screen support can be provided for both tutoring by casestudies 106 and tutoring by procedures 104, as shown in block 110. Ifon-screen support is not desired, the user can indicate whether thetutoring session is actually an official test, as shown in block 112.Thus, tutoring system 90 includes both the ability to teach and theability to test the student.

According to a preferred embodiment of the present invention, thetutorial system also includes a simplified version of the simulatedendoscopic process for teaching the proper manipulation of the endoscopeaccording to visual feedback, as well as for enabling the student tounderstand the correspondence between the visual feedback and tactilefeedback. This simplified version would emphasize the performance andmastery of one or more specific tasks, such as the manipulation of theendoscope through the colon.

Indeed, this preferred embodiment could be generalized to a method forteaching a particular skill required for performance of an actualmedical procedure to a student. This method would include the step ofabstracting a portion of the visual feedback of the actual medicalprocedure, which would preferably include fewer visual details than theentirety of the visual feedback obtained during the performance of themedical procedure. This portion of the visual feedback would preferablyenable the student to learn the motion of the instrument as the requiredskill.

For example, the simplified version may optionally not feature many, oreven most, of the visual details of the colon as visual feedback.Instead, the colon would preferably be presented as a smooth, relativelyfeatureless tube having the geometry and dimensions of the colon inorder to correlate the motion of the simulated endoscope through theinterior space of the colon. More preferably, the simplified versionwould be embodied as a game, in which students would be awarded pointsfor correct manipulation of the endoscope, and would be penalized forincorrect manipulations. Thus, the student would have the opportunity tolearn the manipulations required for successful endoscopy without thedistraction of visual details, in a low pressure and even “fun”environment.

FIGS. 5A and 5B illustrate the mechanical aspects of an exemplarysimulated gastro-intestinal tract according to the present invention. Acut-away view of a mannequin 114 is shown in FIG. 5A. Preferably,mannequin 114 is about one meter wide, which is within the dimensions ofan actual human subject. A simulated gastro-intestinal tract 116 isshown within mannequin 114. For the purposes of clarity, simulatedgastrointestinal tract 116 includes only the colon, it being understoodthat this is not meant to be limiting in any way. Simulatedgastro-intestinal tract 116 is connected to a transmitter 118 and asignal processing device 120, also placed within mannequin 114. Asshown, a simulated endoscope 122 can be inserted into mannequin 114through an opening 124. In this case, since the simulation is forendoscopy of the colon of the subject, opening 124 simulates the rectumof the subject.

Simulated endoscope 122 can be maneuvered left, right, up and down.Preferably, simulated endoscope 122 is about 1800 cm long, similar tothe length of a real endoscope. Also preferably, the diameter of the tipof simulated endoscope 122 is about 13.4 mm, while the remainder ofendoscope 122 has a diameter of about 10.2 mm, again similar to thedimensions of a real endoscope.

Once simulated endoscope 122 is inserted into simulatedgastro-intestinal tract 116, sensor 76 on the tip of simulated endoscope122 is able to detect the location of simulated endoscope 122. Sensor 76preferably has three degrees of freedom, more preferably six degrees offreedom for effective simulation of manipulation of endoscope 122. Ifsensor 76 has six degrees of freedom, the detected directions oforientation include the Cartesian coordinates X, Y, Z, as well as roll,elevation and azimuth. In addition, sensor 76 preferably includes asensor transmitter 126, so that the precise angle and location of sensor76 can be determined relative to gastrointestinal tract 116. Sensortransmitter 126 transmits data to signal processing device 120, whichthen analyzes and processes the signal. The processed signal is thengiven to transmitter 118 for transmission to an electronics unit 128 anda DC drive unit 130. The signal is converted by DC drive unit 130 andpassed to electronics unit 128. Electronics unit 128 then sends theposition and orientation of sensor 76 to software interface 74, so thatthe remainder of the display system is able to use the information todisplay the correct images on display screen 22 for visual feedback.

The present invention provides both visual feedback and tactilefeedback. Tactile feedback can be provided through the exertion offorces on simulated endoscope 122 by simulated gastro-intestinal tract116, as shown in FIGS. 6A-6C. Alternatively, tactile feedback could beprovided by the mechanical action of simulated endoscope 122, as shownin FIGS. 7A-7D. For the first embodiment, preferably simulatedgastro-intestinal tract 116 is constructed from semi-flexible material,which gives the feel of a smooth and wet material. Of course, the actualsensations of sliding along a semi-flexible, smooth, wet material canalso be provided through the mechanism of endoscope 122 itself, as inthe second embodiment.

An additional embodiment of gastro-intestinal tract 116, in which tract116 is placed within a box 132 rather than within mannequin 114, isshown in FIG. 5B. The advantage of box 132 is that box 132 could serveto contain any radiowaves, so that the mechanism of gastro-intestinaltract 116 could be controlled by transmission of radiowaves, forexample. Since certain medical equipment is highly sensitive to theseradiowaves, they would need to remain within mannequin 114. Box 132would therefore act to insulate gastro-intestinal tract 116 from theexternal environment outside the mannequin. Details of gastrointestinaltract 116 are more readily seen in FIG. 6A, it being understood thatFIGS. 5A, 5B and 6A illustrate the same gastrointestinal tract 116.

FIG. 6A shows gastro-intestinal tract 116 according to the firstembodiment, in which tactile feedback is provided by forces acting onsimulated endoscope 122 by a mechanism contained within gastrointestinaltract 116 itself. Simulated gastro-intestinal tract 116 is made from asemi-flexible material. A plurality of motion boxes 134 are disposed atintervals along the outer surface of gastro-intestinal tract 116. Forthe purposes of illustration, seven motion boxes 134 are shown. Eachmotion box 134, shown in greater detail in FIG. 6B, has at least one,and preferably a plurality of, servo-motors 80, preferably linearmotors.

Each servo-motor 80 is connected to a piston 136. The detail of piston136 is shown enlarged in FIG. 6B. Each piston 136 is connected to a foot138, which contacts a portion of the material of the external surface ofgastrointestinal tract 116. Preferably, foot 138 is actually attached tothe portion of the material of the external surface, for easiermanipulation of the material.

Preferably, there are two different types of pistons 136. The firsttype, of which two are shown for illustrative purposes, is a verticalforce piston 140 for causing vertical movement of a portion of theexternal surface of gastro-intestinal tract 116. The second type, ofwhich one is shown for illustrative purposes, is a horizontal forcepiston 142 for causing horizontal movement of a portion of the externalsurface of gastro-intestinal tract 116. In the preferred embodimentshown, servomotor 80 is an oscillating motor placed directly against thematerial of gastro-intestinal tract 116, so that horizontal force piston142 includes the motor alone, without a structure similar to verticalforce piston 140. Since each piston 136 has an associated servo-motor80, the necessary vertical and horizontal movement of the externalsurface of gastro-intestinal tract 116 can be precisely determined bythe activity of servo-motor 80.

Each piston 136, or preferably attached foot 138, contacts the materialof gastro-intestinal tract 116 in order to manipulate this material toexert a force against the endoscope (not shown). For example, as shownin FIG. 6B, a first vertical force piston 144 could be moved closer toservo-motor 80, while a second vertical force piston 146 is moved awayfrom servo-motor 80. These movements alter the position of the materialof gastro-intestinal tract 116, causing forces to be exerted against thesimulated endoscope similar or identical to those felt during an actualendoscopic procedure. In addition, horizontal force piston 142, which ispreferably an oscillating servo-motor alone as shown, moves horizontallyto provide more delicate fine-tuning of the tactile feedback sensations.Since servo-motors 80 are disposed over the three-dimensional surface ofgastrointestinal tract 116, the force on the endoscope can be exerted inthree dimensions.

The activity of servo-motor 80 is in turn controlled by digitalcontroller 82. Digital controller 82 can be a card inserted within thePC computer which is performing the requisite calculations required forthe simulation of the medical process. Software operated by the PCcomputer uses positional and orientation information from sensor 76 onsimulated endoscope 122 to determine the position of simulated endoscope122. Next, the software sends instructions to digital controller 82according to the desired tactile sensations which should be felt by theoperator of simulated endoscope 122 at that particular position withinsimulated gastrointestinal tract 116. Digital controller 82 then causesat least one servo-motor 80 to move the associated piston 136 asnecessary to provide the tactile feedback sensations.

Digital controller 82 can be connected to servo-motors 80 through sometype of radiation, such as infra-red light. However, the limitations onradiation of certain wavelengths, such as radiowaves, within thehospital or medical environment, make a connection by an actual wirerunning from digital controller 82 to each servo-motor 80 morepreferable. In the exemplary embodiment shown in FIG. 6B, eachservo-motor 80 is connected to a motion box controller 144 by a wire.Motion box controller 144 is then preferably connected to digitalcontroller 82 by a single wire (not shown). This configuration limitsthe number of individual connections made to digital controller 82 forgreater efficiency.

FIG. 6C shows an enlarged cut-away view of servo-motor 80, which asnoted previously is preferably a linear motor. Preferably, servo-motor80 is about 100 mm wide and 45 mm tall.

FIGS. 7A-7D show a second embodiment of the mechanism for providingtactile feedback. In this embodiment, the mechanism is contained withinthe simulated endoscope itself, rather than the simulatedgastrointestinal tract. Similar to the previous embodiment, thesimulated gastrointestinal tract could be contained within asubstantially life-size mannequin with an opening for simulating therectum. Furthermore, from the viewpoint of the student or otherindividual operating the simulated endoscope, both embodiments shouldgive a suitable simulation of the medical procedure. However, asdetailed below, the actual mechanism of providing the tactile portion ofthe simulation differs.

FIG. 7A shows the second embodiment of a simulated endoscope 146. Themovements and actions of simulated endoscope 146 are controlled througha set of controls 148. The tip of simulated endoscope 146 is containedwithin a guiding sleeve 150. Guiding sleeve 150, shown in greater detailin FIG. 7B, preferably remains within the simulated gastrointestinaltract (not shown; see FIG. 7C) in order to maintain a realistic visualappearance of simulated endoscope 146 before insertion into themannequin (not shown). Preferably, the tip of endoscope 146 has a metalbracket 152 attached, which could be labeled with the word “sample” orwith another label in order to clarify that endoscope 146 is only asimulation and not an actual medical instrument. The inside of guidingsleeve 150 is preferably magnetized, for example with an electriccurrent. Thus, when the tip of endoscope 146 is inserted in themannequin, metal bracket 152 is attracted to guiding sleeve 150 so thatguiding sleeve 150 remains attached to the tip of endoscope 146.

Guiding sleeve 150 has at least one, and preferably a plurality of, ballbearings 154 attached to the exterior surface of guiding sleeve 150. Inaddition, guiding sleeve 150 has at least one, and preferably aplurality of, attached plungers 156. As shown in the detailed view inFIG. 7B, one end of guiding sleeve 150 preferably features a section offlexible material 158. As shown, the tip of endoscope 146 is preferablyinserted through guiding sleeve 150, The tip of endoscope 146 featuressensor 76, as for the previous embodiment of the simulated endoscope.

FIG. 7C shows simulated endoscope 146 after insertion within the secondembodiment of a simulated gastro-intestinal tract 160. Simulatedgastro-intestinal tract 160 is preferably constructed from a rigidmaterial. In addition, simulated gastro-intestinal tract 160 preferablyhas the general anatomical shape and features of an actualgastro-intestinal tract for two reasons. First, the general anatomicalshape can be more easily contained within the mannequin because of itsbends and turns. Second, the general anatomical shape can provide grosstactile feedback. For example, as any endoscope is inserted more deeplyinto the colon, the shape of the colon causes the tactile sensations tobe altered as the endoscope moves around a bend in the colon. Thus, thegeneral anatomical shape is more useful for an effective simulation.

As endoscope 146 moves within simulated gastro-intestinal tract 160,guiding sleeve 150 enables the operator to receive tactile feedback asfollows. Ball bearings 154 roll along the interior surface ofgastrointestinal tract 160. Each ball bearing 154 has five degrees offreedom for movement. Each plunger 156 is connected to a linear motor162, as shown in cross-section in FIG. 7D. Linear motor 162 iscontrolled in a similar fashion as the servo-motor of the previousembodiment. Upon receipt of signals from the computer, linear motor 162causes plunger 156 to move vertically, thereby causing the operator ofsimulated endoscope 146 to receive tactile feedback sensations. Thus,guiding sleeve 150 causes tactile feedback to be transmitted backthrough endoscope 146.

In addition, as noted above guiding sleeve 150 preferably has section offlexible material 158. Section of flexible material 158 causes the tipof endoscope 146 to encounter some resistance under certaincircumstances, such as when the tip is bent back on itself. Thus,section of flexible material 158 restrains movement of the tip fromcertain angles.

The particular advantages of this second embodiment is that the majorityof tactile sensations are determined by the endoscope itself, so thatthey can be more easily controlled from the PC computer. Furthermore,such anatomical features as a fistula can be added according toinstructions from the computer, without the necessity of changing thephysical model of the simulated gastro-intestinal tract. Additionally,under certain circumstances the tissue of the actual colon will forcethe endoscope backwards, a situation which can be more easily replicatedin the second embodiment. Thus, the second embodiment of the simulatedgastro-intestinal tract and endoscope is more flexible in terns ofreplicating a greater variety of anatomical features and conditions.

FIGS. 8A-8E show yet another and particularly preferred embodiment ofthe simulated endoscope and colon according to the present invention.FIG. 8A shows a preferred system for medical simulation according to thepresent invention. A system 164 includes a mannequin 166 representingthe subject on which the procedure is to be performed, a simulatedendoscope (not shown, see FIG. 8D) and a computer 168 with a videomonitor 170. Mannequin 166 preferably includes a palpable area 172 fordetermining the location of the simulated endoscope by feeling theabdominal area of mannequin 166. Palpable area 172 preferably features alight (not shown), such that when the student has determined thelocation of the simulated endoscope, the light is lit to show the actuallocation of the simulated endoscope.

Mannequin 166 also includes a simulated organ 174 into which thesimulated endoscope is inserted. Preferably, simulated organ 174 is acolon, which more preferably is constructed as a straight tube, with theforce feedback required for the curves in the colon provided through aforce feedback mechanism 176. More preferably, the visual feedback forthe simulated medical procedure does not depend upon the geometricalshape of simulated organ 174 itself, such that the visual feedback andthe tactile feedback are both substantially completely independent ofthe construction of simulated organ 174.

Force feedback mechanism 176 preferably includes an air-driven forcefeedback device 178 (shown in more detail in FIGS. 8B, 8D and 8E). Morepreferably, two such air-driven force feedback devices 178 are provided,one near a mouth 180 of mannequin 166, and the other near a rectum 182of mannequin 166. An air tube 184 connects each air-driven forcefeedback device 178 to an air-pump 186. Preferably, air-pump 186 alsoincludes an air-pump control unit 188 which is connected to computer 168for controlling the amount of air pumped into air-driven force feedbackdevice 178.

Computer 168 also preferably includes a modem 190 for communication withother computers. For example, modem 190 could enable computer 168 toconnect to the Internet or intranet for performing telemedicine, or toconnect to the intranet/computer network of the manufacturer for repairor trouble-shooting.

FIGS. 8B and 8C show components of air-driven force feedback device 178in more detail. As shown in FIG. 8B, a portion of a simulated endoscope192 interacts with air-driven force feedback device 178 to provide forcefeedback to the student. Force feedback device 178 features a pluralityof inflatable rings 194 (shown in more detail in the fully inflatedposition in FIG. 8C). Each inflatable ring 194 preferably has adifferent radius. More preferably, there are four such rings 194, atleast one of which has a larger radius than endoscope 192 and at leastone of which has a smaller radius than endoscope 192. The amount of airfed into rings 194 determines the degree of inflation of each ring 194,preferably separately, thereby determining the amount of force exertedonto endoscope 192.

Preferably, each ring 194 requires one second or more preferably lessthan one second to reach the fully inflated position. The air flow rateis preferably up to 100 liters per minute and the pressure is up to 3atmospheres. Rings 194 are preferably used both for passive forcefeedback, such as from the contraction of the rectum, and for activeforce feedback, for example when air is pumped into simulated organ 174according to a functional feature of simulated endoscope 192 (see FIG.8E).

FIG. 8D shows force feedback mechanism 176 in more detail. Preferably,rings 194 are connected to air pump 186 through tube 184, which morepreferably is split into two tubes 196, a first tube 196 for pumping airinto rings 194, and a second tube 196 for pumping air from rings 194.The amount of air pumped by air pump 186 is controlled by air pumpcontroller 188. The actions of air pump controller 188 are preferablycontrolled by computer 168 through an I/O (analog-to-digital) card 198.

FIG. 8E shows simulated endoscope 192 in more detail. Simulatedendoscope 192 features a handle 200 with various controls, including afirst control 202 for pumping air into simulated organ 174, and a secondcontrol 204 for suctioning air out of simulated organ 174. Simulatedendoscope 192 preferably features a surgical tool control device 206into which various surgical tools are optionally and preferably inserted(see FIGS. 9A-9E). Simulated endoscope 192 also preferably features areceiver 208, for example a “minibird” sensor (Ascension Ltd.,Burlington, Vt., USA). Receiver 208 is located at the tip of simulatedendoscope 192. Receiver 208 is designed to receive transmissions from atransmitter 210 located in mannequin 166 (see FIG. 8A), therebydetermining a position of the tip of simulated endoscope 192 withinsimulated organ 174. Transmitter 210 is preferably a “minibird”transmitter (Ascension Ltd.). Receiver 208 then transmits these signalsto computer 168, which uses these signals for determining the amount offorce feedback and the visual feedback to be displayed to the student onmonitor 178.

As previously described, FIGS. 9A-9E show a preferred implementation ofsurgical tool control device 206 into which various surgical tools areoptionally and preferably inserted. Surgical tool control device 206preferably features a forceps 212 inserted into a tool sleeve 214,thereby simulating actual forceps for an endoscope. Actual forceps areused for performing a polypectomy, and feature a loop which emerges fromthe tip of the forceps upon manipulation of the device. This loop isplaced around the polyp and drawn tight. Electricity is then sentthrough the loop in order to cut the polyp and to cauterize the area.

Similar to actual forceps, forceps 212 is inserted as the student holdsa forceps handle 216, preferably including a button or other control forsimulating the effects of starting the flow of “electricity” through the“loop”. Tool sleeve 214 features a tool control unit 218 for detectingthe motions of forceps 212, and translating these motions into forcefeedback and visual feedback. Visual feedback includes the visualdisplay of the forceps “loop” when appropriate, for example, as well asthe display of the polyp before and after the “polypectomy”. Inaddition, the location of the loop must be tracked, preferably includingup and down movements within the endoscope, and “roll” movement of theloop. Tool control unit 218 is connected to an I/O card within thecomputer (not shown) for performing the necessary calculations for thevarious types of feedback.

FIGS. 9B and 9C show two views of forceps 212 interacting with toolcontrol unit 218 within tool sleeve 214. Tool control unit 218 featuresa guide wheel 220 and a light wheel 222 for detecting the motions offorceps 212 (FIG. 9B). Light wheel 222 features a plurality of notchesthrough which light may pass. Tool control unit 218 also features afirst light 224 and a first light sensor 226, as well as a second light228 and a second light sensor 230 (FIG. 9C). As light wheel 222 turnswith the motion of forceps 212, light passing from first light 224 andsecond light 228 is alternately blocked and unblocked, such that lightis alternately detectable and non-detectable by first light sensor 226and second light sensor 230.

FIG. 9C shows a second embodiment of the tool control unit. In thisembodiment, a tool control unit 232 features two guide wheels 234. Guidewheels 234 help to guide the movement of forceps 212 within tool sleeve214. A light wheel 236 also features notches through which light isalternately blocked and unblocked as forceps 212 is rotated within toolsleeve 214. A light source (not shown) produces light which is detected,if it passes through light wheel 236, by a photoelectric eye 238.Photoelectric eye 238 then sends signals to a PCB (printed circuitboard) 240 which is connected to the computer (not shown), such thatthese signals can be translated by the computer into the required visualfeedback and force feedback.

A foot pedal 242 is shown in FIG. 9E for performing a simulatedpolypectomy. Foot pedal 242 features an oil piston 244 and a microswitch246. Microswitch 246 is connected to an I/O card on the computer (notshown), again for translating the movement of foot pedal 242 into therequired visual feedback and force feedback.

In order to accurately replicate the tactile sensations of an actualendoscope during a medical procedure, these sensations must beaccurately obtained during an endoscopic procedure in an actual livingpatient. For example, such tactile sensations could be collected from aphysician performing the endoscopic procedure while wearing virtualreality gloves, such as the DataGloves™ Tracking VR System (GreenleafMedical Systems). These gloves are known for being able to register dataregarding tactile sensations and feedback as experienced by thephysician during the actual endoscopic procedure. Such actual data areimportant because the tactile sensations change during the course of theprocedure. For example, correlation between the movement of theendoscope and the visual display is gradually decreased as the endoscopeis inserted deeper into the gastrointestinal tract. Thus, the collectionof actual data is an important step in the provision of an accurate,realistic endoscopic simulator.

Finally, according to another preferred embodiment of the presentinvention there is provided a simulated biopsy device (not shown). Thisbiopsy device would simulate the actual biopsy device used to retrievetissue samples from the gastro-intestinal tract during endoscopy. Theactual biopsy device is contained within the endoscope. When theoperator of the endoscope wishes to take a sample, the biopsy deviceemerges from the tip of the endoscope, at which point it is visible onthe display screen. The jaws of the biopsy device are then opened andpushed onto the tissue. The jaws are then closed, and the biopsy deviceretracted. The removal of the tissue causes pools of blood to appear asthe remaining tissue bleeds.

Similarly, the simulated biopsy device will only appear on the displayscreen of the present invention when the operator of the simulatedendoscope causes the simulated biopsy device to emerge. The jaws of thebiopsy device are preferably rendered as animation, more preferably inrelatively high resolution because the jaws are small, so that a highresolution would not prove unduly taxing for the PC computer. Thebleeding of the tissue and the resultant pools of blood will also beanimated.

It will be appreciated that the above descriptions are intended only toserve as examples, and that many other embodiments are possible withinthe spirit and the scope of the present invention.

1. A system for performing a simulated medical procedure, comprising:(a) a simulated organ, wherein said simulated organ is agastro-intestinal tract; (b) a simulated instrument for performing thesimulated medical procedure on said simulated organ, wherein saidsimulated instrument is an endoscope, said endoscope featuring a sensorfor determining a location of said sensor in said gastro-intestinaltract; (c) a locator for determining a location of said simulatedinstrument within said simulated organ; (d) a visual display fordisplaying images according to said location of said simulatedinstrument within said simulated organ for providing visual feedback,such that said images simulate actual visual data received during anactual medical procedure as performed on an actual subject, said visualdisplay including: (i) a three-dimensional mathematical model formodeling said simulated organ according to a corresponding actual organ,said model being divided into a plurality of segments, said plurality ofsegments being arranged in a linear sequence; (ii) a loader forselecting at least one of said plurality of segments from said linearsequence for display, said at least one of said plurality of segmentsbeing selected according to said location of said simulated instrumentwithin said simulated organ; (iii) a controller for selecting asimulated image from said segment according to said location of saidsimulated instrument; and (iv) a displayer for displaying said simulatedimage; (e) a computer for determining said visual feedback according tosaid location of said sensor; and (f) a tactile feedback mechanism forproviding simulated tactile feedback according to said location of saidtip of said endoscope, wherein said tactile feedback mechanism islocated in said endoscope, and said endoscope further comprises: (i) aguiding sleeve connected to said tip of said endoscope; (ii) at leastone ball bearing attached to said guiding sleeve for rolling along aninner surface of said gastro-intestinal tract; (iii) at least one motorattached to said guiding sleeve; (iv) a piston operated by said linearmotor, said piston contacting said inner surface of saidgastro-intestinal tract; and (v) a controller for controlling saidlinear motor, such that a position of said piston is determined by saidcontroller, and such that said position of said piston provides saidtactile feedback.
 2. A system for performing a simulated medicalprocedure, comprising: (a) a simulated organ; (b) a simulated instrumentfor performing the simulated medical procedure on said simulated organ,said simulated instrument featuring a sensor for determining a locationof said sensor; (c) a locator for determining a location of a tip ofsaid simulated instrument within said simulated organ according to saidsensor; (d) a visual display for displaying images according to saidlocation of said simulated instrument within said simulated organ forproviding visual feedback, such that said images simulate actual visualdata received during an actual medical procedure as performed on anactual subject, said visual display including: (i) a three-dimensionalmathematical model for modeling said simulated organ according to acorresponding actual organ, said model being divided into a plurality ofsegments, said plurality of segments being arranged in a linearsequence; (ii) a loader for selecting at least one of said plurality ofsegments from said linear sequence for display, said at least one ofsaid plurality of segments being selected according to said location ofsaid simulated instrument within said simulated organ; (iii) acontroller for selecting a simulated image from said segment accordingto said location of said simulated instrument; and (iv) a displayer fordisplaying said simulated image; and (e) a tactile feedback mechanismfor providing simulated tactile feedback according to said location ofsaid tip of said endoscope, wherein said tactile feedback mechanismfeatures; (i) a plurality of rings surrounding said endoscope, each ringhaving a different radius, at least a first ring featuring a radiusgreater than a radius of said endoscope and at least a second ringfeaturing a radius less than said radius of said endoscope, said radiusof each of said plurality of rings being controlled according to adegree of inflation with air of each of said plurality of rings, saidradius of said rings determining movement of said endoscope; (ii) an airpump for pumping air into said plurality of rings; (iii) at least onetube for connecting said air pump to said plurality of rings; and (iv)an air pump controller for determining said degree of inflation with airof said plurality of rings by controlling said air pump.
 3. A system forperforming a simulated medical procedure, comprising: (a) a simulatedorgan; (b) a simulated instrument for performing the simulated medicalprocedure on said simulated organ, wherein said simulated instrument isan endoscope featuring an endoscope cable, said endoscope cable forminga loop from a movement of said endoscope in said simulated organ; (c) alocator for determining a location of said simulated instrument withinsaid simulated organ; and (d) a visual display for displaying imagesaccording to said location of said simulated instrument within saidsimulated organ for providing visual feedback, such that said imagessimulate actual visual data received during an actual medical procedureas performed on an actual subject, said visual display including: (i) athree-dimensional mathematical model for modeling said simulated organaccording to a corresponding actual organ, said model being divided intoa plurality of segments, said plurality of segments being arranged in alinear sequence, wherein said mathematical model features a plurality ofpolygons defined with respect to a spline, said spline determining ageometry of said mathematical model in three dimensions, said loop beingmodeled according to said mathematical model, wherein said mathematicalmodel for said loop features a plurality of polygons defined withrespect to a spline, and wherein a size of said loop is determinedaccording to a differential between an amount of said endoscope cablewithin said simulated organ and a length of said simulated organ from anentry point of said endoscope to a current position of said endoscopewithin said simulated organ; (ii) a loader for selecting at least one ofsaid plurality of segments from said linear sequence for display, saidat least one of said plurality of segments being selected according tosaid location of said simulated instrument within said simulated organ;(iii) a controller for selecting a simulated image from said segmentaccording to said location of said simulated instrument; and (iv) adisplayer for displaying said simulated image.
 4. A system forsimulating a medical procedure, the system comprising: (a) an instrumentfor being manipulated for performing the simulated medical procedure;(b) a three-dimensional mathematical model of an organ, such that avirtual location of said instrument in the organ during the simulatedmedical procedure is determined according to said three-dimensionalmathematical model, wherein said mathematical model features a spline,said spline determining a geometry of said mathematical model in threedimensions; (c) a visual display for providing visual feedback accordingto said virtual location and said three-dimensional mathematical model;and (d) a tactile feedback mechanism for providing simulated tactilefeedback according to said virtual location of said instrument; whereinsaid instrument is an endoscope featuring an endoscopic cable, saidendoscope cable forming a loop from a movement of said endoscope in theorgan, said loop being modeled according to a mathematical model,wherein said mathematical model for said loop features a plurality ofpolygons defined with respect to a spline, and wherein a size of saidloop is determined according to a differential between an amount of saidendoscope cable within the organ and a length of the organ from an entrypoint of said endoscope to said virtual location of said endoscopewithin the organ.